Observations  from  AT&L/PARCA's 

Root  Cause  Analyses 

NPS  Acquisition  Symposium 
May  17th,  2012 


Gary  R.  Bliss 
Director,  PARCA 


Report  Documentation  Page 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 

1.  REPORT  DATE 

17  MAY  2012  2' REPORT  TYPE 

3.  DATES  COVERED 

00-00-2012  to  00-00-2012 

4.  TITLE  AND  SUBTITLE 

Observations  from  AT&L/PARCA’s  Root  Cause  Analyses 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Office  of  the  Assistant  Secretary  of  Defense  for  Acquisition, Performance 
Assessments  and  Root  Cause  Analyses  (PARCA),3620  Defense  Pentagon 
RM  5A1082, Washington, DC, 20301-3620 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS (ES) 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

Presented  at  the  9th  Annual  Acquisition  Research  Symposium,  May  16  - 17 

,  2012,  Monterey,  CA. 

14.  ABSTRACT 

15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF:  17.  LIMITATION  OF 

ARSTRATT 

1 8 .  NUMBER  1 9a.  NAME  OF 

OF  PAGES  RESPONSIBLE  PERSON 

a.  REPORT  b.  ABSTRACT  c.  THIS  PAGE  Same  3S 

unclassified  unclassified  unclassified  Report  (SAR) 

24 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


Overview 


►  What  is  PARCA’s  role? _ 

►  Steps  to  improve  baseline  estimates 

►  Other  actions 
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Root  Cause  Analysis  Functions 

►  Statutory  duties  defined  in  WSARA  09 

-  Conduct  root  cause  analyses  for  major  defense  acquisition 
programs. 

■  As  part  of  the  Nunn-McCurdy  breach  certification  process. 

WSARA  09  states  that  the  Secretary  of  Defense  shall 

(1)  determine  the  root  cause  or  causes  of  the  critical  cost  growth 

(2)  If  program  is  not  terminated,  restructure  the  program  in  a 
manner  that  addresses  root  cause  or  causes  of  the  critical 
cost  growth 

■  When  requested  by  designated  officials. 

-  Issue  policies,  procedures,  and  guidance  governing  the  conduct 
of  root  cause  analyses. 

►  Identification  of  lessons  learned  for  the  benefit 
of  acquisition  community. 
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Problems  will  occur:  why  they  occur  and  our  response 
to  them  are  crucial  subjects  for  root  cause  analysis. 
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Why  are  Estimates  Unrealistic? 
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►  Unrealistic  estimates  are  generally  caused  by  the  invalidity 
of  major  assumptions  NOT  methodological  errors. 


Orbiter  Processing  Facility  Concept  (1974)  Actual  Orbiter  Processing  Facility 

►  The  cost  estimating  community  can  and  should  challenge 
assumptions  but  the  acquisition  community  formulates  them. 

►  Consideration  of  this  has  led  to  “framing  assumptions” 


Estimating  Assumptions  Flow  from 
Framing  Assumptions 


Framing  Assumptions 


Design  is  mature 

(Prototype  design  is  close  to  Production-Ready) 


Consequences 


Estimating  Assumptions 


Responsible  Communities: 

Requirements,  Technical, 

&  Program  Management 

Cost  Estimators 


Production  and 
development  can  be 
concurrent 

Weight  (critical  for 
vertical  lift)  is  known 

Design  can  now  be 
refined  for  affordability 

+  + 

Schedule  will  be  more 
compact  than  historical 
experience 

Weight  will  not  grow 
as  usual  for  tactical 
aircraft 

Affordability  initiatives 
will  reduce  production 
cost 

Cost  and  Schedule  Estimates 
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Illustrative  Sources  for  Framing 

Assumptions 


Pre-MS  B  activities:  The  design  is  very  similar  to  the  ACTD. 

Technical  base:  Modular  construction  will  result  in  significant 
cost  savings. 

Policy  implementation:  The  conditions  are  met  for  a  firm,  fixed 

_  _Pri??  contract _ 

Organizational:  Arbitrating  multi-Service  requirements  will  be 
straightforward. 

Program  dependencies:  FCS  will  facilitate  solution  of  size, 
weight,  and  power  issues.  Interoperability 

Threat  or  operational  needs:  The  need  for  precision  strike  of 
urban  targets  will  not  decline. 

Industrial  base/market:  The  satellite  bus  will  have  substantial 
commercial  market  for  the  duration  of  program. 
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Framing  Assumptions  and  Decision-Making 


►  Issues  now  are  formed  by  “advocates”  of  particular  subject 

areas: 

-  Each  subject  area  has  particular  metrics 

-  Relationship  of  these  to  overall  program  success  is  never  questioned 

►  Intent  is  to  raise  the  key  issues  for  the  program  irrespective 

of  whether  they  are  controversial 

-  First  step:  Identify  the  right  issues  and  know  how  they  contribute  to 
program  success. 

-  Second  step:  Establish  what  metrics  are  relevant  to  the  issue’s 
contribution  to  program  success. 

-  Third  step:  Present  the  data  to  date  for  and  against,  including  relevant 
historical  programs  that  are  capable  of  discriminating  outcomes. 

-  Fourth  step:  Generate  baseline  forecasts  of  how  the  data  will  evolve  if 
the  thesis  is  correct . . .  And  vice  versa.  Track  data  and  report. 
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Poor  Management  Performance 


>  Management  performance  is 
the  lens  through  which  all 
program  issues  are  addressed. 

-  Contractor 

-  Program  Office 
-PEO 

-OSD 


>  PARCA  has  found  issues  in  three  broad  areas. 

-  Systems  engineering 

-  Contractual  incentives 

-  Organizational  awareness  and  response 

A  critical  but  difficult  area  to  assess. 
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Systems  Engineering 


►  General  observations. 


-  High  potential  as  a  significant  or  a  root  cause  because 
systems  engineering  critical  for  complex  systems. 

-  Process  definition  required  but  can  dominate  substance. 

“Strong  in  process  but  weak  in  integration”  -  PMO  assessment 

-  Recognizing  poor  systems  engineering  early  a  challenge. 
-“Systems  Engineering”  too  broad  for  actionable  root  causes. 


►  Areas  where  problems  have  been  observed  include: 


-Requirements  management 

■  Ambiguities  in  combining  requirements  documents. 

■  Funding  program  to  include  all  requirements. 

-Interface  and  environment  management 
-Holistic  performance  attributes  e.g.  reliability,  weight 


APPROVED  FOR  PUBLIC  RELEASE  DISTRIBUTION  UNLIMITED:  12-8-0915, 27  JAN  2012. 


11 


Effective  Contracting  Strategy 


►  Incentive  evaluation 

-Aligned  with  program  goals  and 
challenges 

-Demanding  yet  achievable 

-  Sufficient  to  motivate 

-  No  perverse  effects 
-Correct  signal  sent  and  received 

►  Incentive  strategy 


Year 


-Conditions  for  strategy  satisfied? 

-Consistent  with  corporate  goals  and  position 
-Consistent  with  policy _ 

Government  goals  but  contractor’s  perspective 
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Situational  Awareness  -  Contract 
Performance  v  Program  Performance 


►  Several  cases  where 

program  content  evolves 
-Work  not  understood 
-“Unfunded”  requirement 
-Spiral  development  (almost  by 
definition) 

►  Consequences  require 
management 


Breach 


-Contract  performance  differs  from 
program  performance 
-Discipline  in  program  content 
-  Budgetary  pressure 
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Closing  Comments 


►  PARCA’s  role  in  N-M 
process  is  to  help  identify 
program  root  causes  so  that 
we  don’t  have  more  growth 

►  The  broader  issue  is  -  can 
the  breach  be  avoided 
altogether? 

►  PARCA’s  observations 
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Percent  cost  growth 


-  Problem  cost  growth  comes  both  from  inception  and  execution  issues 

-  Inception  issues  are  primarily  due  to  invalid  postulates 

-  In  execution,  DoD  does  not  always  recognize  all  the  implications  of 
postulate  invalidity  and  finds  it  difficult  to  identify  and  respond  to 
evidence  of  invalid  postulates  and  organizational  issues. 

Recognizing  problems  is  the  first  step  to  solving  them. 


APPROVED  FOR  PUBLIC  RELEASE  DISTRIBUTION  UNLIMITED:  12-8-0915, 27  JAN  2012. 


14 


Backup 
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Organizational  Response 


►  Direction 

-  Policy  is  a  significant  source 
of  program  postulates 
-Implementation  not  assured 
e.g.  PM  tenure,  ADMs 

►  Organizational 
performance  issues 
appear  to  persist 


Goal  from  USD(AT&L) 


1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008 


Year  Assigned 


-Organizational  problems  identified  at  first  breach  ... 

■  “Management  and  systems  engineering  processes  were  not  in  place 

■  Contractor  “did  not  effectively  tap  ...  expertise  resident  within  team.” 

■  “Concurrent  activities  increased  program  execution  complexity.” 


-  ...  are  often  still  present  at  second. 
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Correlation  When  Framing 
Assumption  is  Invalid 
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Estimating  Assumptions 


Schedule  will  be  more 
compact  than  historical 
experience 


Responsible  Communities: 

Requirements,  Technical, 

&  Program  Management 

Cost  Estimators 


Weight  will  not  grow 
as  usual  for  tactical 
aircraft 


Affordability  initiatives 
will  reduce  production 
cost 


Cost  and  Schedule  Estimates 
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Role  in  N-M  Breaches 


►  When  an  invalid  framing  assumption  is  embraced: 
-  Evidence  of  problems  will  accumulate 
-Cost  and  schedule  estimates  will  need  to  be  changed 
-But,  the  amount  of  growth  will  depend  on 

■  How  promptly  management  recognizes  the  issues 

■  How  effectively  management  responds 


►  Further  cost  growth  if  the 
full  implications  of  the 
invalid  framing  assumption 
are  not  addressed. 


2000 


2005 


2010 


2015 


“It  ain't  what  you  don't  know  that  gets  you  into  trouble.  It's 
what  you  know  for  sure  that  just  ain't  so.”  -  Mark  Twain 
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Constant  set  of  Assumptions  for  a 
Variety  of  Decisions 


(Prototypes 

Design  is  mature 
demonstrated  key  issue/trades) 

Government 
requirements  have 
been  defined 


Confidence  in  system 
definition 


Integration  will  be 
straightforward 


Resolution  of  SWaP 
issues 


Conditions  for  FFP 
EMD  are  satisfied 


approach  to  contract 
management 


Concept  will  be  piloted  this  year 
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Overview 


►  Unrealistic  Estimates 

►  Changes  in  Quantity 

►  Performance 
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Quantity  Changes 


►  To  consider  a  quantity  change  to  be  a  root  cause, 
PARCA  has  defined  two  conditions: 

-  The  reason  for  the  change  was  outside  the  control  of  the 
acquisition  community. 

■  Doctrinal  or  threat  change  hut  NOT  "  Escalating  unit  costs 

■  “Pure”  fiscal  constraints  "  Schedule  slips 

-  Other  cost  growth  would  not  have  caused  a  breach  without  the 
quantity  change. 

►  To  date,  PARCA  has  observed  that  quantity  changes 
were  due  to  factors  within  the  control  of  the  acquisition 
community  in  about  half  of  the  cases. 


The  reason  for  the  quantity  change  is  crucial  to  management. 
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Preliminary  Trends 
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Root  causes  must  be  true  AND  relevant. 
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The  Impact  of  Changing  Quantity 
Depends  on  Other  Decisions 


►  Quantity  change  in  isolation 
mainly  impacts  PAUC. 

-  RDT&E  amortized  over  fewer 
units. 

-  Modest  learning  effects  on 
APUC. 

►  Two  separate  decisions 
often  affect  unit  costs  as 
much  as  the  quantity 
change  itself. 


Number  of  Units 


-  Reduce  annual  buy  or  curtail  program. 

-  The  mix  of  systems  can  change  (or  confuse)  unit  costs. 
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